System logging (syslog) is a widely used mechanism for remotely sending notification “log” messages from a device to a receiver. The receiver is a logging host and/or management application. Syslog messaging is inherently unreliable because it is cumbersome and difficult for the logging host and/or management application to be assured that it is getting the messages it needs and at the same time not getting inundated with messages that it does not need, which may adversely impact performance of the logging host and/or management application or the device associated with the logging host and/or management application.
To address the issue of receiving only desired messages, a number of techniques have been attempted in the industry. For example, a sequence number is may be added to each syslog message to uniquely identify each syslog message. Sequence numbers permit receiving applications to detect whether messages are lost by detecting sequence number gaps between different syslog messages. A sequence number also permits an application to uniquely identify a particular syslog message across multiple applications; this can be useful if multiple applications compare and merge their logs to create a single non-duplicated log.
Another technique for improving the accuracy of receiving only desired messages is to have the sender of the syslog messages enforce rate limiting or enforce message filtering. With rate limiting, if a rate limit for receiving messages is exceeded, then messages are either queued or dropped altogether. This is done to avoid swamping the receiver, which may cause messages to be dropped anyway by the device associated with the receiver. Similarly, messages may be weeded out such that only messages of interest to an application are sent to the application. In this manner, the quality and quantity of messages may be somewhat controlled by the receivers or applications.
However, in most cases a sender of the syslog messages is delivering syslog messages to multiple receivers where some receivers are enforcing rate limits or filters that are different from the other receivers. Thus, because different messages get dropped for different receivers problems can arise. Specifically, gaps in sequence numbers for received messages may not be definitively identified as problematic because receivers are unable to distinguish whether the gaps are legitimate because of rate limiting or filtering or whether the gaps are in fact problematic because of lost messages that are not properly received from the sender.
Furthermore, if an attempt is made to number syslog messages based on an identity of the receiver in order to alleviate the problem of detecting legitimate from problematic gaps in syslog message numbering, then this creates other problems. For example, if the syslog messages are uniquely numbered for a specific receiver and not globally across all receivers, then multiple receivers can not merge their logs with one another because the uniqueness of a single syslog message across multiple receivers is lost or is undetectable to the receivers.
Therefore, improved syslog message sequencing techniques are desirable.